{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "copyright-notice"
   },
   "source": [
    "#### Copyright 2017 Google LLC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "copyright-notice2",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "# https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mPa95uXvcpcn"
   },
   "source": [
    " # 使用神经网络对手写数字进行分类"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Fdpn8b90u8Tp"
   },
   "source": [
    " ![img](https://www.tensorflow.org/versions/r0.11/images/MNIST.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "c7HLCm66Cs2p"
   },
   "source": [
    " **学习目标：**\n",
    "  * 训练线性模型和神经网络，以对传统 [MNIST](http://yann.lecun.com/exdb/mnist/) 数据集中的手写数字进行分类\n",
    "  * 比较线性分类模型和神经网络分类模型的效果\n",
    "  * 可视化神经网络隐藏层的权重"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HSEh-gNdu8T0"
   },
   "source": [
    " 我们的目标是将每个输入图片与正确的数字相对应。我们会创建一个包含几个隐藏层的神经网络，并在顶部放置一个归一化指数层，以选出最合适的类别。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2NMdE1b-7UIH"
   },
   "source": [
    " ## 设置\n",
    "\n",
    "首先，我们下载数据集、导入 TensorFlow 和其他实用工具，并将数据加载到 *Pandas* `DataFrame`。请注意，此数据是原始 MNIST 训练数据的样本；我们随机选择了 20000 行。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "test": {
      "output": "ignore",
      "timeout": 600
     }
    },
    "colab_type": "code",
    "id": "4LJ4SD8BWHeh",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "import glob\n",
    "import math\n",
    "import os\n",
    "\n",
    "from IPython import display\n",
    "from matplotlib import cm\n",
    "from matplotlib import gridspec\n",
    "from matplotlib import pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "from sklearn import metrics\n",
    "import tensorflow as tf\n",
    "from tensorflow.python.data import Dataset\n",
    "\n",
    "tf.logging.set_verbosity(tf.logging.ERROR)\n",
    "pd.options.display.max_rows = 10\n",
    "pd.options.display.float_format = '{:.1f}'.format\n",
    "\n",
    "mnist_dataframe = pd.read_csv(\n",
    "  \"https://download.mlcc.google.cn/mledu-datasets/mnist_train_small.csv\",\n",
    "  sep=\",\",\n",
    "  header=None)\n",
    "\n",
    "# Use just the first 10,000 records for training/validation.\n",
    "mnist_dataframe = mnist_dataframe.head(10000)\n",
    "\n",
    "mnist_dataframe = mnist_dataframe.reindex(np.random.permutation(mnist_dataframe.index))\n",
    "mnist_dataframe.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kg0-25p2mOi0"
   },
   "source": [
    " 第一列中包含类别标签。其余列中包含特征值，每个像素对应一个特征值，有 `28×28=784` 个像素值，其中大部分像素值都为零；您也许需要花一分钟时间来确认它们不*全部*为零。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PQ7vuOwRCsZ1"
   },
   "source": [
    " ![img](https://www.tensorflow.org/versions/r0.11/images/MNIST-Matrix.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dghlqJPIu8UM"
   },
   "source": [
    " 这些样本都是分辨率相对较低、对比度相对较高的手写数字图片。`0-9` 这十个数字中的每个可能出现的数字均由唯一的类别标签表示。因此，这是一个具有 10 个类别的多类别分类问题。\n",
    "\n",
    "现在，我们解析一下标签和特征，并查看几个样本。注意 `loc` 的使用，借助 `loc`，我们能够基于原来的位置抽出各列，因为此数据集中没有标题行。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "test": {
      "output": "ignore",
      "timeout": 600
     }
    },
    "colab_type": "code",
    "id": "JfFWWvMWDFrR",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "def parse_labels_and_features(dataset):\n",
    "  \"\"\"Extracts labels and features.\n",
    "  \n",
    "  This is a good place to scale or transform the features if needed.\n",
    "  \n",
    "  Args:\n",
    "    dataset: A Pandas `Dataframe`, containing the label on the first column and\n",
    "      monochrome pixel values on the remaining columns, in row major order.\n",
    "  Returns:\n",
    "    A `tuple` `(labels, features)`:\n",
    "      labels: A Pandas `Series`.\n",
    "      features: A Pandas `DataFrame`.\n",
    "  \"\"\"\n",
    "  labels = dataset[0]\n",
    "\n",
    "  # DataFrame.loc index ranges are inclusive at both ends.\n",
    "  features = dataset.loc[:,1:784]\n",
    "  # Scale the data to [0, 1] by dividing out the max value, 255.\n",
    "  # 通过除以最大值255 使得数据缩放到[0,1]之间\n",
    "  features = features / 255\n",
    "\n",
    "  return labels, features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "mFY_-7vZu8UU",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "training_targets, training_examples = parse_labels_and_features(mnist_dataframe[:7500])\n",
    "training_examples.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "4-Vgg-1zu8Ud",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "validation_targets, validation_examples = parse_labels_and_features(mnist_dataframe[7500:10000])\n",
    "validation_examples.describe()\n",
    "#validation_targets.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wrnAI1v6u8Uh"
   },
   "source": [
    " 显示一个随机样本及其对应的标签。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "s-euVJVtu8Ui",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "rand_example = np.random.choice(training_examples.index)\n",
    "_, ax = plt.subplots()\n",
    "# print(training_examples.loc[rand_example].values.reshape(28, 28))\n",
    "ax.matshow(training_examples.loc[rand_example].values.reshape(28, 28))\n",
    "ax.set_title(\"Label: %i\" % training_targets.loc[rand_example])\n",
    "ax.grid(False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ScmYX7xdZMXE"
   },
   "source": [
    " ## 任务 1：为 MNIST 构建线性模型\n",
    "\n",
    "首先，我们创建一个基准模型，作为比较对象。`LinearClassifier` 可提供一组 *k* 类一对多分类器，每个类别（共 *k* 个）对应一个分类器。\n",
    "\n",
    "您会发现，除了报告准确率和绘制对数损失函数随时间变化情况的曲线图之外，我们还展示了一个[**混淆矩阵**](https://en.wikipedia.org/wiki/Confusion_matrix)。混淆矩阵会显示错误分类为其他类别的类别。哪些数字相互之间容易混淆？\n",
    "\n",
    "另请注意，我们会使用 `log_loss` 函数跟踪模型的错误。不应将此函数与用于训练的 `LinearClassifier` 内部损失函数相混淆。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "cpoVC4TSdw5Z",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "def construct_feature_columns():\n",
    "  \"\"\"Construct the TensorFlow Feature Columns.\n",
    "\n",
    "  Returns:\n",
    "    A set of feature columns\n",
    "  \"\"\" \n",
    "  \n",
    "  # There are 784 pixels in each image. \n",
    "  return set([tf.feature_column.numeric_column('pixels', shape=784)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kMmL89yGeTfz"
   },
   "source": [
    " 在本次练习中，我们会对训练和预测使用单独的输入函数，并将这些函数分别嵌套在 `create_training_input_fn()` 和 `create_predict_input_fn()` 中，这样一来，我们就可以调用这些函数，以返回相应的 `_input_fn`，并将其传递到 `.train()` 和 `.predict()` 调用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "OeS47Bmn5Ms2",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "def create_training_input_fn(features, labels, batch_size, num_epochs=None, shuffle=True):\n",
    "  \"\"\"A custom input_fn for sending MNIST data to the estimator for training.\n",
    "\n",
    "  Args:\n",
    "    features: The training features.\n",
    "    labels: The training labels.\n",
    "    batch_size: Batch size to use during training.\n",
    "\n",
    "  Returns:\n",
    "    A function that returns batches of training features and labels during\n",
    "    training.\n",
    "  \"\"\"\n",
    "  def _input_fn(num_epochs=None, shuffle=True):\n",
    "    # Input pipelines are reset with each call to .train(). To ensure model\n",
    "    # gets a good sampling of data, even when number of steps is small, we \n",
    "    # shuffle all the data before creating the Dataset object\n",
    "    idx = np.random.permutation(features.index)\n",
    "    raw_features = {\"pixels\":features.reindex(idx)}\n",
    "    raw_targets = np.array(labels[idx])\n",
    "   \n",
    "    ds = Dataset.from_tensor_slices((raw_features,raw_targets)) # warning: 2GB limit\n",
    "    ds = ds.batch(batch_size).repeat(num_epochs)\n",
    "    \n",
    "    if shuffle:\n",
    "      ds = ds.shuffle(10000)\n",
    "    \n",
    "    # Return the next batch of data.\n",
    "    feature_batch, label_batch = ds.make_one_shot_iterator().get_next()\n",
    "    return feature_batch, label_batch\n",
    "\n",
    "  return _input_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "8zoGWAoohrwS",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "def create_predict_input_fn(features, labels, batch_size):\n",
    "  \"\"\"A custom input_fn for sending mnist data to the estimator for predictions.\n",
    "\n",
    "  Args:\n",
    "    features: The features to base predictions on.\n",
    "    labels: The labels of the prediction examples.\n",
    "\n",
    "  Returns:\n",
    "    A function that returns features and labels for predictions.\n",
    "  \"\"\"\n",
    "  def _input_fn():\n",
    "    raw_features = {\"pixels\": features.values}\n",
    "    raw_targets = np.array(labels)\n",
    "    \n",
    "    ds = Dataset.from_tensor_slices((raw_features, raw_targets)) # warning: 2GB limit\n",
    "    ds = ds.batch(batch_size)\n",
    "    \n",
    "        \n",
    "    # Return the next batch of data.\n",
    "    feature_batch, label_batch = ds.make_one_shot_iterator().get_next()\n",
    "    return feature_batch, label_batch\n",
    "\n",
    "  return _input_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "G6DjSLZMu8Um",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "def train_linear_classification_model(\n",
    "    learning_rate,\n",
    "    steps,\n",
    "    batch_size,\n",
    "    training_examples,\n",
    "    training_targets,\n",
    "    validation_examples,\n",
    "    validation_targets):\n",
    "  \"\"\"Trains a linear classification model for the MNIST digits dataset.\n",
    "  \n",
    "  In addition to training, this function also prints training progress information,\n",
    "  a plot of the training and validation loss over time, and a confusion\n",
    "  matrix.\n",
    "  \n",
    "  Args:\n",
    "    learning_rate: An `int`, the learning rate to use.\n",
    "    steps: A non-zero `int`, the total number of training steps. A training step\n",
    "      consists of a forward and backward pass using a single batch.\n",
    "    batch_size: A non-zero `int`, the batch size.\n",
    "    training_examples: A `DataFrame` containing the training features.\n",
    "    training_targets: A `DataFrame` containing the training labels.\n",
    "    validation_examples: A `DataFrame` containing the validation features.\n",
    "    validation_targets: A `DataFrame` containing the validation labels.\n",
    "      \n",
    "  Returns:\n",
    "    The trained `LinearClassifier` object.\n",
    "  \"\"\"\n",
    "\n",
    "  periods = 10\n",
    "\n",
    "  steps_per_period = steps / periods  \n",
    "  # Create the input functions.\n",
    "  predict_training_input_fn = create_predict_input_fn(\n",
    "    training_examples, training_targets, batch_size)\n",
    "  predict_validation_input_fn = create_predict_input_fn(\n",
    "    validation_examples, validation_targets, batch_size)\n",
    "  training_input_fn = create_training_input_fn(\n",
    "    training_examples, training_targets, batch_size)\n",
    "  \n",
    "  # Create a LinearClassifier object.\n",
    "  my_optimizer = tf.train.AdagradOptimizer(learning_rate=learning_rate)\n",
    "  my_optimizer = tf.contrib.estimator.clip_gradients_by_norm(my_optimizer, 5.0)\n",
    "  classifier = tf.estimator.LinearClassifier(\n",
    "      feature_columns=construct_feature_columns(),\n",
    "      n_classes=10,\n",
    "      optimizer=my_optimizer,\n",
    "      config=tf.estimator.RunConfig(keep_checkpoint_max=1)\n",
    "  )\n",
    "\n",
    "  # Train the model, but do so inside a loop so that we can periodically assess\n",
    "  # loss metrics.\n",
    "  print(\"Training model...\")\n",
    "  print(\"LogLoss error (on validation data):\")\n",
    "  training_errors = []\n",
    "  validation_errors = []\n",
    "  for period in range (0, periods):\n",
    "    # Train the model, starting from the prior state.\n",
    "    classifier.train(\n",
    "        input_fn=training_input_fn,\n",
    "        steps=steps_per_period\n",
    "    )\n",
    "  \n",
    "    # Take a break and compute probabilities.\n",
    "    training_predictions = list(classifier.predict(input_fn=predict_training_input_fn))\n",
    "    training_probabilities = np.array([item['probabilities'] for item in training_predictions])\n",
    "    training_pred_class_id = np.array([item['class_ids'][0] for item in training_predictions])\n",
    "    training_pred_one_hot = tf.keras.utils.to_categorical(training_pred_class_id,10)\n",
    "        \n",
    "    validation_predictions = list(classifier.predict(input_fn=predict_validation_input_fn))\n",
    "    validation_probabilities = np.array([item['probabilities'] for item in validation_predictions])    \n",
    "    validation_pred_class_id = np.array([item['class_ids'][0] for item in validation_predictions])\n",
    "    validation_pred_one_hot = tf.keras.utils.to_categorical(validation_pred_class_id,10)    \n",
    "    \n",
    "    # Compute training and validation errors.\n",
    "    training_log_loss = metrics.log_loss(training_targets, training_pred_one_hot)\n",
    "    validation_log_loss = metrics.log_loss(validation_targets, validation_pred_one_hot)\n",
    "    # Occasionally print the current loss.\n",
    "    print(\"  period %02d : %0.2f\" % (period, validation_log_loss))\n",
    "    # Add the loss metrics from this period to our list.\n",
    "    training_errors.append(training_log_loss)\n",
    "    validation_errors.append(validation_log_loss)\n",
    "  print(\"Model training finished.\")\n",
    "  # Remove event files to save disk space.\n",
    "  _ = map(os.remove, glob.glob(os.path.join(classifier.model_dir, 'events.out.tfevents*')))\n",
    "  \n",
    "  # Calculate final predictions (not probabilities, as above).\n",
    "  final_predictions = classifier.predict(input_fn=predict_validation_input_fn)\n",
    "  final_predictions = np.array([item['class_ids'][0] for item in final_predictions])\n",
    "  \n",
    "  \n",
    "  accuracy = metrics.accuracy_score(validation_targets, final_predictions)\n",
    "  print(\"Final accuracy (on validation data): %0.2f\" % accuracy)\n",
    "\n",
    "  # Output a graph of loss metrics over periods.\n",
    "  plt.ylabel(\"LogLoss\")\n",
    "  plt.xlabel(\"Periods\")\n",
    "  plt.title(\"LogLoss vs. Periods\")\n",
    "  plt.plot(training_errors, label=\"training\")\n",
    "  plt.plot(validation_errors, label=\"validation\")\n",
    "  plt.legend()\n",
    "  plt.show()\n",
    "  \n",
    "  # Output a plot of the confusion matrix.\n",
    "  cm = metrics.confusion_matrix(validation_targets, final_predictions)\n",
    "  # Normalize the confusion matrix by row (i.e by the number of samples\n",
    "  # in each class).\n",
    "  cm_normalized = cm.astype(\"float\") / cm.sum(axis=1)[:, np.newaxis]\n",
    "  ax = sns.heatmap(cm_normalized, cmap=\"bone_r\")\n",
    "  ax.set_aspect(1)\n",
    "  plt.title(\"Confusion matrix\")\n",
    "  plt.ylabel(\"True label\")\n",
    "  plt.xlabel(\"Predicted label\")\n",
    "  plt.show()\n",
    "\n",
    "  return classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ItHIUyv2u8Ur"
   },
   "source": [
    " **花费 5 分钟的时间了解一下使用这种形式的线性模型时，准确率方面表现如何。在本次练习中，为自己设定限制，仅使用批量大小、学习速率和步数这三个超参数进行试验。**\n",
    "\n",
    "如果您从上述任何试验中得到的准确率约为 0.9，即可停止试验。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "yaiIhIQqu8Uv",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "classifier = train_linear_classification_model(\n",
    "             learning_rate=0.02,\n",
    "             steps=100,\n",
    "             batch_size=10,\n",
    "             training_examples=training_examples,\n",
    "             training_targets=training_targets,\n",
    "             validation_examples=validation_examples,\n",
    "             validation_targets=validation_targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "266KQvZoMxMv"
   },
   "source": [
    " ### 解决方案\n",
    "\n",
    "点击下方即可查看一种可能的解决方案。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lRWcn24DM3qa"
   },
   "source": [
    " 以下是一组使准确率应该约为 0.9 的参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "TGlBMrUoM1K_",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "_ = train_linear_classification_model(\n",
    "    learning_rate=0.03,\n",
    "    steps=1000,\n",
    "    batch_size=30,\n",
    "    training_examples=training_examples,\n",
    "    training_targets=training_targets,\n",
    "    validation_examples=validation_examples,\n",
    "    validation_targets=validation_targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mk095OfpPdOx"
   },
   "source": [
    " ## 任务 2：使用神经网络替换线性分类器\n",
    "\n",
    "**使用 [`DNNClassifier`](https://www.tensorflow.org/api_docs/python/tf/estimator/DNNClassifier) 替换上面的 LinearClassifier，并查找可实现 0.95 或更高准确率的参数组合。**\n",
    "\n",
    "您可能希望尝试 Dropout 等其他正则化方法。这些额外的正则化方法已记录在 `DNNClassifier` 类的注释中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "rm8P_Ttwu8U4",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "#\n",
    "# YOUR CODE HERE: Replace the linear classifier with a neural network.\n",
    "#"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TOfmiSvqu8U9"
   },
   "source": [
    " 获得出色的模型后，通过评估我们将在下面加载的测试数据进行仔细检查，确认您没有过拟合验证集。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "evlB5ubzu8VJ",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "mnist_test_dataframe = pd.read_csv(\n",
    "  \"https://download.mlcc.google.cn/mledu-datasets/mnist_test.csv\",\n",
    "  sep=\",\",\n",
    "  header=None)\n",
    "\n",
    "test_targets, test_examples = parse_labels_and_features(mnist_test_dataframe)\n",
    "test_examples.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "PDuLd2Hcu8VL",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "#\n",
    "# YOUR CODE HERE: Calculate accuracy on the test set.\n",
    "#"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6sfw3LH0Oycm"
   },
   "source": [
    " ### 解决方案\n",
    "\n",
    "点击下方即可查看可能的解决方案。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XatDGFKEO374"
   },
   "source": [
    " 除了神经网络专用配置（例如隐藏单元的超参数）之外，以下代码与原始的 `LinearClassifer` 训练代码几乎完全相同。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "kdNTx8jkPQUx",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "def train_nn_classification_model(\n",
    "    learning_rate,\n",
    "    steps,\n",
    "    batch_size,\n",
    "    hidden_units,\n",
    "    training_examples,\n",
    "    training_targets,\n",
    "    validation_examples,\n",
    "    validation_targets):\n",
    "  \"\"\"Trains a neural network classification model for the MNIST digits dataset.\n",
    "  \n",
    "  In addition to training, this function also prints training progress information,\n",
    "  a plot of the training and validation loss over time, as well as a confusion\n",
    "  matrix.\n",
    "  \n",
    "  Args:\n",
    "    learning_rate: An `int`, the learning rate to use.\n",
    "    steps: A non-zero `int`, the total number of training steps. A training step\n",
    "      consists of a forward and backward pass using a single batch.\n",
    "    batch_size: A non-zero `int`, the batch size.\n",
    "    hidden_units: A `list` of int values, specifying the number of neurons in each layer.\n",
    "    training_examples: A `DataFrame` containing the training features.\n",
    "    training_targets: A `DataFrame` containing the training labels.\n",
    "    validation_examples: A `DataFrame` containing the validation features.\n",
    "    validation_targets: A `DataFrame` containing the validation labels.\n",
    "      \n",
    "  Returns:\n",
    "    The trained `DNNClassifier` object.\n",
    "  \"\"\"\n",
    "\n",
    "  periods = 10\n",
    "  # Caution: input pipelines are reset with each call to train. \n",
    "  # If the number of steps is small, your model may never see most of the data.  \n",
    "  # So with multiple `.train` calls like this you may want to control the length \n",
    "  # of training with num_epochs passed to the input_fn. Or, you can do a really-big shuffle, \n",
    "  # or since it's in-memory data, shuffle all the data in the `input_fn`.\n",
    "  steps_per_period = steps / periods  \n",
    "  # Create the input functions.\n",
    "  predict_training_input_fn = create_predict_input_fn(\n",
    "    training_examples, training_targets, batch_size)\n",
    "  predict_validation_input_fn = create_predict_input_fn(\n",
    "    validation_examples, validation_targets, batch_size)\n",
    "  training_input_fn = create_training_input_fn(\n",
    "    training_examples, training_targets, batch_size)\n",
    "  \n",
    "  # Create the input functions.\n",
    "  predict_training_input_fn = create_predict_input_fn(\n",
    "    training_examples, training_targets, batch_size)\n",
    "  predict_validation_input_fn = create_predict_input_fn(\n",
    "    validation_examples, validation_targets, batch_size)\n",
    "  training_input_fn = create_training_input_fn(\n",
    "    training_examples, training_targets, batch_size)\n",
    "  \n",
    "  # Create feature columns.\n",
    "  feature_columns = [tf.feature_column.numeric_column('pixels', shape=784)]\n",
    "\n",
    "  # Create a DNNClassifier object.\n",
    "  my_optimizer = tf.train.AdagradOptimizer(learning_rate=learning_rate)\n",
    "  my_optimizer = tf.contrib.estimator.clip_gradients_by_norm(my_optimizer, 5.0)\n",
    "  classifier = tf.estimator.DNNClassifier(\n",
    "      feature_columns=feature_columns,\n",
    "      n_classes=10,\n",
    "      hidden_units=hidden_units,\n",
    "      optimizer=my_optimizer,\n",
    "      config=tf.contrib.learn.RunConfig(keep_checkpoint_max=1)\n",
    "  )\n",
    "\n",
    "  # Train the model, but do so inside a loop so that we can periodically assess\n",
    "  # loss metrics.\n",
    "  print(\"Training model...\")\n",
    "  print(\"LogLoss error (on validation data):\")\n",
    "  training_errors = []\n",
    "  validation_errors = []\n",
    "  for period in range (0, periods):\n",
    "    # Train the model, starting from the prior state.\n",
    "    classifier.train(\n",
    "        input_fn=training_input_fn,\n",
    "        steps=steps_per_period\n",
    "    )\n",
    "  \n",
    "    # Take a break and compute probabilities.\n",
    "    training_predictions = list(classifier.predict(input_fn=predict_training_input_fn))\n",
    "    training_probabilities = np.array([item['probabilities'] for item in training_predictions])\n",
    "    training_pred_class_id = np.array([item['class_ids'][0] for item in training_predictions])\n",
    "    training_pred_one_hot = tf.keras.utils.to_categorical(training_pred_class_id,10)\n",
    "        \n",
    "    validation_predictions = list(classifier.predict(input_fn=predict_validation_input_fn))\n",
    "    validation_probabilities = np.array([item['probabilities'] for item in validation_predictions])    \n",
    "    validation_pred_class_id = np.array([item['class_ids'][0] for item in validation_predictions])\n",
    "    validation_pred_one_hot = tf.keras.utils.to_categorical(validation_pred_class_id,10)    \n",
    "    \n",
    "    # Compute training and validation errors.\n",
    "    training_log_loss = metrics.log_loss(training_targets, training_pred_one_hot)\n",
    "    validation_log_loss = metrics.log_loss(validation_targets, validation_pred_one_hot)\n",
    "    # Occasionally print the current loss.\n",
    "    print(\"  period %02d : %0.2f\" % (period, validation_log_loss))\n",
    "    # Add the loss metrics from this period to our list.\n",
    "    training_errors.append(training_log_loss)\n",
    "    validation_errors.append(validation_log_loss)\n",
    "  print(\"Model training finished.\")\n",
    "  # Remove event files to save disk space.\n",
    "  _ = map(os.remove, glob.glob(os.path.join(classifier.model_dir, 'events.out.tfevents*')))\n",
    "  \n",
    "  # Calculate final predictions (not probabilities, as above).\n",
    "  final_predictions = classifier.predict(input_fn=predict_validation_input_fn)\n",
    "  final_predictions = np.array([item['class_ids'][0] for item in final_predictions])\n",
    "  \n",
    "  \n",
    "  accuracy = metrics.accuracy_score(validation_targets, final_predictions)\n",
    "  print(\"Final accuracy (on validation data): %0.2f\" % accuracy)\n",
    "\n",
    "  # Output a graph of loss metrics over periods.\n",
    "  plt.ylabel(\"LogLoss\")\n",
    "  plt.xlabel(\"Periods\")\n",
    "  plt.title(\"LogLoss vs. Periods\")\n",
    "  plt.plot(training_errors, label=\"training\")\n",
    "  plt.plot(validation_errors, label=\"validation\")\n",
    "  plt.legend()\n",
    "  plt.show()\n",
    "  \n",
    "  # Output a plot of the confusion matrix.\n",
    "  cm = metrics.confusion_matrix(validation_targets, final_predictions)\n",
    "  # Normalize the confusion matrix by row (i.e by the number of samples\n",
    "  # in each class).\n",
    "  cm_normalized = cm.astype(\"float\") / cm.sum(axis=1)[:, np.newaxis]\n",
    "  ax = sns.heatmap(cm_normalized, cmap=\"bone_r\")\n",
    "  ax.set_aspect(1)\n",
    "  plt.title(\"Confusion matrix\")\n",
    "  plt.ylabel(\"True label\")\n",
    "  plt.xlabel(\"Predicted label\")\n",
    "  plt.show()\n",
    "\n",
    "  return classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "ZfzsTYGPPU8I",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "classifier = train_nn_classification_model(\n",
    "    learning_rate=0.05,\n",
    "    steps=1000,\n",
    "    batch_size=30,\n",
    "    hidden_units=[100, 100],\n",
    "    training_examples=training_examples,\n",
    "    training_targets=training_targets,\n",
    "    validation_examples=validation_examples,\n",
    "    validation_targets=validation_targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qXvrOgtUR-zD"
   },
   "source": [
    " 接下来，我们来验证测试集的准确率。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "scQNpDePSFjt",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "mnist_test_dataframe = pd.read_csv(\n",
    "  \"https://download.mlcc.google.cn/mledu-datasets/mnist_test.csv\",\n",
    "  sep=\",\",\n",
    "  header=None)\n",
    "\n",
    "test_targets, test_examples = parse_labels_and_features(mnist_test_dataframe)\n",
    "test_examples.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "EVaWpWKvSHmu",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "predict_test_input_fn = create_predict_input_fn(\n",
    "    test_examples, test_targets, batch_size=100)\n",
    "\n",
    "test_predictions = classifier.predict(input_fn=predict_test_input_fn)\n",
    "test_predictions = np.array([item['class_ids'][0] for item in test_predictions])\n",
    "  \n",
    "accuracy = metrics.accuracy_score(test_targets, test_predictions)\n",
    "print(\"Accuracy on test data: %0.2f\" % accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WX2mQBAEcisO"
   },
   "source": [
    " ## 任务 3：可视化第一个隐藏层的权重。\n",
    "\n",
    "我们来花几分钟时间看看模型的 `weights_` 属性，以深入探索我们的神经网络，并了解它学到了哪些规律。\n",
    "\n",
    "模型的输入层有 `784` 个权重，对应于 `28×28` 像素输入图片。第一个隐藏层将有 `784×N` 个权重，其中 `N` 指的是该层中的节点数。我们可以将这些权重重新变回 `28×28` 像素的图片，具体方法是将 `N` 个 `1×784` 权重数组*变形*为 `N` 个 `28×28` 大小数组。\n",
    "\n",
    "运行以下单元格，绘制权重曲线图。请注意，此单元格要求名为 \"classifier\" 的 `DNNClassifier` 已经过训练。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "test": {
      "output": "ignore",
      "timeout": 600
     }
    },
    "colab_type": "code",
    "id": "eUC0Z8nbafgG",
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "print(classifier.get_variable_names())\n",
    "\n",
    "weights0 = classifier.get_variable_value(\"dnn/hiddenlayer_0/kernel\")\n",
    "\n",
    "print(\"weights0 shape:\", weights0.shape)\n",
    "\n",
    "num_nodes = weights0.shape[1]\n",
    "num_rows = int(math.ceil(num_nodes / 10.0))\n",
    "fig, axes = plt.subplots(num_rows, 10, figsize=(20, 2 * num_rows))\n",
    "for coef, ax in zip(weights0.T, axes.ravel()):\n",
    "    # Weights in coef is reshaped from 1x784 to 28x28.\n",
    "    ax.matshow(coef.reshape(28, 28), cmap=plt.cm.pink)\n",
    "    ax.set_xticks(())\n",
    "    ax.set_yticks(())\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kL8MEhNgrx9N"
   },
   "source": [
    " 神经网络的第一个隐藏层应该会对一些级别特别低的特征进行建模，因此可视化权重可能只显示一些模糊的区域，也可能只显示数字的某几个部分。此外，您可能还会看到一些基本上是噪点（这些噪点要么不收敛，要么被更高的层忽略）的神经元。\n",
    "\n",
    "在迭代不同的次数后停止训练并查看效果，可能会发现有趣的结果。\n",
    "\n",
    "**分别用 10、100 和 1000 步训练分类器。然后重新运行此可视化。**\n",
    "\n",
    "您看到不同级别的收敛之间有哪些直观上的差异？"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "266KQvZoMxMv",
    "6sfw3LH0Oycm",
    "copyright-notice"
   ],
   "default_view": {},
   "name": "multi-class_classification_of_handwritten_digits.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
